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Abstract: The research project has developed a design framework for an adaptive MOOC that complements the MOOC 
format with blended learning. The design framework consists of a design model and a series of learning design principles 
which can be used to design in-service courses for teacher professional development. The framework has been evaluated 
by alpha-testing and beta-testing, and the relationship between design principles and the intended, the implemented and 
the attained designs has been analyzed. The project is methodologically inspired by Design Based Research. 
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1 Introduction 

In 2020 it will be a requirement that Danish primary school teachers have a bachelor degree in the subjects 
they teach. More than 10,000 teachers, who for many years have taught a course without being formally 
qualified, need professional development and therefore municipalities ask for new concepts for in-service 
training. There is a need for concepts for supplementary training that are flexible in relation to teachers' work 
situations, are based on the fact that the teachers already have a number of professional skills, and at the 
same time are resource-efficient compared to the price and the time teachers must use to be formally 
qualified. Finally, the concept has to be scalable because it is uncertain how many teachers need training 
within each subject area. A number of municipalities (the customers) and University College Zealand (UCSJ: the 
provider) are in the process of examining whether the new training format "MOOC" (Massive Open Online 
Course) can solve this training task. 

As part of this process UCSJ has established a research project with the aim of developing a design framework 
which can guide the development of instructional designs, adapted to experienced teachers' different learning 
needs and study the factors affecting the actual realization, legitimacy and efficacy of the design. 

2 Methods 

The project is methodologically inspired by Design Based Research (DBR) (Brown, 1992; Collins, 1992) - a 
method widely used in MOOC research (Gasevic et al, 2014). The design framework has been developed 
through iterative design experiments (diSessa & Cobb, 2004). Several prototypes have been evaluated and 
redesigned and research has generated theory that can guide the further development process. We have 
analyzed interviews with participants and teachers and made observations of the participants' interactions 
with each other and with the technology (Moodle). Through these design experiments it has been possible to 
develop a design framework consisting of a set of pedagogical design principles that can be communicated 
through one or more design examples (Hrastinski et al, 2010). 

3 Prior research 

Research in MOOCs has until 2012 been scarce (Kennedy, 2014; Liyanagunawardena et al, 2013), which is of no 
surprise since MOOCs were first offered in 2007/2008. 

MOOCs are based on previous research in e-learning (King, 2014) and experience from Open Educational 
Resources (OER) where MOOCs differ by integrating OER in an instructional design with an embedded but 
mediated teacher presence and clear learning objectives (Liyanagunawardena et al, 2013). The research has 
been oriented towards only a few topics including MOOC typologies with the dichotomy: C-MOOCs to X- 
MOOCs (Rodriguez, 2012; Liyanagunawardena et al, 2013). Rodriguez describes a C-MOOC as an online course 
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that don't align with the course content nor the instructor, but to the learners and their knowledge. They are 
build inspired by the philosophy of "Connectivism" (Rodriguez, 2012). X-MOOCs are based on interactive 
media, primarily videos and text. X-MOOCs are typically run by a corporate education company and have 
adopted a more behaviourist pedagogical approach, with the emphasis on individual learning, rather than 
learning through peers (Conole, 2014). 

However this discussion is not so dominant anymore (Bayne & Ross, 2014), and several MOOC providers are 
developing MOOCs that integrate several different pedagogical approaches depending on the objectives of the 
learning inspired, for example, by Laurillards pedagogical framework (Laurillard, 2012; King et al, 2014). 

Participant perspectives and especially the high dropout rates among students still have had great attention 
(Rodriguez, 2012; Kennedy, 2014; Vivian, 2014). Hypotheses about how to reduce the dropout rate based on 
research show that learning is supported if the participants can interact with each other and with the teacher. 
Social presence and teaching presence, therefore, are important in an educational design (Kop et al., 2011) - 
forms of presence which particularly have been explored in the context 

of the "Community of Inquiry" framework (Col)(Garrison & Anderson, 2011). The Col framework stresses that 
the students educational experience depends on three types of presence: The cognitive presence, which is the 
students engagement with the content, the social presence, which is the students engagement with other 
participants and the teaching presence, which is the teachers engagement in the course (Garrison & Anderson, 
2011). The Col framework stresses that the students educational experience depends on three types of 
presence: The cognitive presence, which is the students engagement with the content, the social presence, 
which is the students engagement with other participants and the teaching presence, which is the teachers 
engagement in the course. 

MOOC research has been primarily oriented towards developing social interaction between participants, for 
example, through peer to peer response methods. However, this has not solved the problem concerning 
dropouts (Gasevic et al, 2014). In formal education, where high dropout rates are unacceptable, there has 
been an increasing interest in educational designs that blend MOOCs with either on-campus teaching or 
synchronous online teaching and learning environments (Bayne & Ross, 2014; Gasevic et al, 2014; Holotescu et 
al, 2014; Israel, 2015). 

The typical MOOC student is an adult who already has a degree and is fully or partly in job. The participation in 
a MOOC is for professional development either out of personal intellectual curiosity or in connection with the 
acquisition of specialized skills related to work (Vivian, 2014; Kellogg 2014; King et al, 2014). MOOCs can be an 
effective design for acquiring work specialized skills if the design is competency-based and enables 
personalized learning that matches the professional's need for additional skills (Norton et al, 2013, Milligan & 
Littlejohn 2014; Gasevic et al, 2014). This requires an educational design that can identify an individual's skills, 
identify skill needs and adaptively design a study for each student (Kostolanyova & Sarmanova, 2014). 
Personalized learning and adaptive education is a growing field of research (Kinshuk, 2015), also within the 
field of MOOCs where research, however, has been limited (Gasevic et al, 2014). The concepts of 
"personalized learning" and "adaptive education" are not clear concepts. "Personalized learning" is used not 
only to characterize competence-oriented learning but also in connection with theories about particular 
learning styles (Akbulut & Cardak, 2012). "Adaptive education" also covers several areas, including adaptive 
technical systems and adaptive instructional designs, better known as differentiated instruction. In the latter 
case, curriculum, learning resources, teaching and guidance are tailored to the learner's needs. 

4 Design framework for adaptive learning design - initial considerations 

Adaption is feedback from an educational system adapted to the needs of learners (Bateson, 1998; Hattie, 
2011). We distinguish between different forms of feedback (U.S. Department of Education, 2010): 

Differentiation is education, where participants have the same learning goals, but the teaching method varies 
so they adapt to the individual student's needs. 


www.ejel.org 


16 


©ACPIL 



Karsten Gynther 


Individualization is teaching, where the participants also have the same learning goals, but participants can 
move forward at different speeds and relate to a particular content area or a given activity in different ways, 
and teaching is tailored to individual needs. 

Personalization is education, where participants have different learning objectives, depending on their learning 
needs. The training is customized, so this is possible, and personalized instruction may also provide 
opportunities for differentiation and individualization. 

In a research field, which grew from an influential study on adaptation by Lee J. Cronbach (Cronbach, 1957), it 
has been documented by many educational researchers that adaptive learning designs which adapt teaching 
to the individual learner's needs, have an positive effect on the learning outcome (Akbulut & Cardak, 2012). 

Attempts to individualize instruction with a technical system is however an older idea. Frederick Taylor (1911) 
was interested in the idea of a "teaching machine". In 1958 B.F. Skinner introduced the idea of technology 
mediated programmed learning (Skinner, 1958), and in the 70s a lot of research in the field of Computer- 
Assisted Instruction (CAI) took place. The criticism of this approach and especially the radical behaviorism that 
Skinner developed has been intense in education research for decades (Wenger, 1998). 

Adaptive learning systems are this century's attempt to develop an educational technology adapted to users' 
needs, and Siemens et al (2015) refers to this technology as "fifth-generation" educational technologies. 
Important knowledge can be gained from the design and development of these systems, even if taking into 
consideration the basic criticism of the learning theories on which some of these systems are based. 

Adaptive learning systems 

There is a variety of different definitions of adaptive learning systems. The differences are mainly related to 
the level of adaptation, one imagines a system must be able to perform in relation to a participant and the 
learning process. 

Most adaptive learning systems consist of three components (Natriello, 2011; Oxman & Wong, 2014): 

■ A content model. 

■ A learner model. 

■ An instructional design model which is a strategy for the adaption process. 

4.1 Content model 

The content model structures the content of learning objectives, sequences and tasks to be solved (Natriello, 
2011). A content model divides the subject into smaller elements, which can be associated with different types 
of learning resources (Thalmann, 2014). 

4.2 Learner model 

An adaptive learning system also contains a model of the learner (Wenger, 1987). The model is based on one 
or both of the following categories: a) the learner's current knowledge, and b) the learner's learning 
preferences. 

It is widely agreed that it has a learning effect if the teaching is adapted to the individual's knowledge. Pre¬ 
understanding or prior knowledge is considered as one of the individual factors that has the greatest 
importance in a learning process (Glaser, 1984). The model of the learner must visualize the personalized 
curriculum a given person should be offered in a concrete course. Most adaptive learning systems therefore 
identify the learner's existing knowledge and compare the learner's knowledge with the knowledge structure 
or curriculum for a given subject. 

The majority of all commercial adaptive learning systems also try to model the learner's preferences for certain 
types of learning processes. Attempts to categorize the learners in cognitive types or learning styles are here 
very common. In a review of 70 published articles on adaptive learning systems (Akbulut & Cardak, 2012) 81 % 
of the participating learning systems were using cognitive types or learning styles for modeling learners. Most 
used were cognitive types based on Kolb (1984) and learning styles based on Felder- Silman (1988) or Dunn 
and Dunn (1974). Despite the widespread use of models of the learner building on typologies of preferences in 
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terms of learning styles or cognitive types, the same study showed that "findings on concrete learning 
outcomes were not strong enough" (Akbulut & Cardak, 2012 s. 835). 

It is therefore important to be critical towards adaptive learning systems that emphasize the identification of 
specific preferences and hypotheses concerning specific learning styles. Especially because the development of 
a model of the learner on the basis of hypotheses related to the learner's preferences can develop into what is 
called "stereotype methods" (Shute & Zapata-Rivera, 2010). 

The third dimension in an adaptive learning system is the strategy of adaptation. In terms of supporting the 
learner's navigation in the system the designers of the system use a variety of adaptation strategies. Basically, 
we can distinguish between two adaptation strategies: recommendation systems and guided navigation (Khribi 
et a I, 2015). In a recommendation system the technology identifies a range of possibilities which the system 
priorities for the learner on the basis of a learner model or on the basis of the learner's performance in the 
system. But the learner is free to choose whether to follow the recommendation. By guided navigation the 
system hides the links which are not relevant to the learner, either because they do not match the model of 
the learner or because they do not match the learner's continuous performance in the system. 

An important design discussion is the question of who should have control of the adaption process. Is it the 
system or the learner (Shute & Zapata-Rivera, 2010; Simens et al, 2015)? Review of research on adaptive 
learning systems shows that this is not always reflected in the design of the adaptive learning system (Akbulut 
& Cardak, 2012; Ford, 2013). The problem is that the adaption process may be invisible to the learner, since 
the rules or algorithms which are used to control the system is not known or understood by the user. The 
system can collect a large amount of data about the user (big data) through the monitoring of learners' 
interactions with the system (Siemens et a I, 2015). This raises a number of ethical questions and dilemmas of 
privacy and users' control of their own data. Who owns the data, an adaptive learning system produces, and 
what can and should this data be used for? 

5 Design criteria 

Based on knowledge about adaptive learning systems, we have defined a set of design criteria for the 
development of adaptive learning design in general: 

■ Modelling of the learner must be based on documented effects. 

■ Development of adaptive learning design must be based on a precautionary principle (ethical code) which 
means that we do not use stereotypical methods for modeling of the learner. 

■ Modeling should (only) visualize a) the learner's professional skills and b) experience and skills to learn in a 
MOOC format. 

■ Adaptation performed by a technical system based on non-transparent algorithms cannot stand alone. 

■ Adaptation must be a dialogue (negotiation) between the learner and a teacher on the basis of one or 
more technically-generated information. 

■ The adaption strategy should be recommendations and the adaptation process must be transparent and 
controlled by the learner. 

■ The learner must control own data. 

6 Findings 

Based on the research review above, a series of design workshops and three iterative design experiments, we 
have developed a design framework for design of adaptive learning environments in formal education. 
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Figure 1: Design framework for an adaptive hybrid MOOC 
The design framework visualizes three design levels: 

6.1 Setting 

The design framework is based on a well-known design model which frames the design as a setting for formal 
training with a participant, a content and a teacher. 

But we are following the widespread criticism of this model and situate the three elements of the model in the 
context they are part of (Garisson & Anderson, 2011). 

■ The learner is part of a personal learning network (PLN). 

■ The subject is part of a broader academic culture and its interpretation of the subject. 

■ The teacher is situated in an educational institution and more widely in an educational system. 

The development of a specific adaptive learning design must be based on the framing and the concrete 

anchorage of the three elements in their specific contexts. 

6.2 Relationships 

The framework visualizes the characteristics of the relationship between the design elements described above: 

■ The relationship between the learner and the subject is characterized by a personalized curriculum. Each 
participant has their own unique curriculum. The project has shown that participants with long working 
experience in a field have acquired a number of competencies related to the curriculum - skills that are 
very different from participant to participant. 

■ The relationship between a participant and the teacher is characterized by complementarity. In traditional 
teaching concepts the relation between participant and a teacher is the core of the instructional design 
and teacher presence is the starting point for concrete designs for learning. However, this is not possible 
in an instructional design where all participants have their own personalized curriculum. In a group of 
participants who each have their own curriculum it is not possible to realize a multiple relationship: a 
participant - a content - a teacher. The relationship between participant and teacher must be 
complementary if you want to support that all participants have a personalized curriculum. 

■ Finally, the relationship between the teacher and the subject also has a characteristic feature that is far 
from usual perceptions about being a teacher. The traditional role of the teacher is the lecturer who 
interprets a subject and mediates the relationship between the learner and the subject in a face-to-face 
setting. The teacher identifies himself with the role of being a teacher. The project shows that the 
relationship between the teacher and the subject must be transformed from a teacher role to an author 
role. The teacher is rather a designer, an author and a producer of a number of learning resources. A role 
that also entails that the teacher is part of a larger production team. 
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6.3 Principles 

Level 3 in the model visualizes the design principles. These principles relate to each of the three characteristics 
described above. 

6.3.1 Personalized curriculum: Multiple learning path: 

The design must be able to: 

■ identify the participants current skills - visualized in a competency profile. 

* visualize a competence-gap in terms of a personalized curriculum. 

■ recommend a learning path which adaptively matches the learner's personalized curriculum. 

■ identify the student's ability to learn in and with a MOOC. 

■ establish an adaptive scaffolding of the student's learning process in the MOOC. 

The principle of multiple learning pathways, which we will refer to as the design potential or affordance of the 
design. 

6.3.2 Production of learning resources: The content model 

In order to realize the principle of multiple pathways of learning, the educational institution in advance has to 
produce a content model of the course, that: 

* covers the entire curriculum of the subject. 

■ includes a deconstruction of the subject to competency units. 

■ guides the production of learning resources and forms of participation, which without progression are 
linked to each unit of competence. 

This design principle can be described as a constraint for adaptive learning designs. The project demonstrates 
that particularly the breakdown of the subject to competencies which participants can study with no 
progression is a major challenge for the teacher/author that produces the concrete MOOCs. 

The design framework includes no constraints regarding the choice of types of activity associated with specific 
stereotypes, learning styles, etc. The framework thus encourages the development of a number of different 
types of activity associated with each competency including: 

■ Passive activities: Participants are exposed to a learning resource. 

■ Active activities: Participants need to do something related to a resource (solve a quiz, etc.). 

* Constructive activities: Participants must produce inputs to the system that contains ideas that are not 
found in an available system resource. (Ex. formulating solutions to a problem linked to their own 
practice). 

■ Interactive activities: Participants are engaged in a dialogue with another participant, a teacher or the 
system on a given subject matter. 

The above list is based on Natriello, 2011 and we will add: 

■ Collaborative activities: Participants create together an output on the basis of the above activities with the 
possibility of including resources and tools that are not available in the learning design. 

6.3.3 Complementary teacher presence: Representation of the teacher 

The final design principle is a key constraint for the design of MOOCs in general and thus also for adaptive 
designs for learning on the basis of the MOOC format. Since the teacher cannot be present in a multiple 
number of learning pathways, the teacher must be represented in the design. The teacher must be mediated 
in a form that minimizes the disadvantage of a learning design where the teacher cannot be physically present. 
The concrete mediation of the teacher should, as far as possible, allow personalization of the teacher even 
though it is not possible to get immediate synchronous feedback from a present teacher. Therefore, we are 
working on developing a design principle which we tentatively call asynchronous teacher telepresence. The 
project shows that it is extremely difficult for teachers to accept this constraint, and therefore educational 
institutions need to scaffold teachers who must perform the transformation from "teacher presence" to 
"asynchronous teacher telepresence". At the same time the project demonstrates that there are many ways to 
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mediate the presence of the teacher including multiple video formats which are well known from MOOCs in 
general. 

The principle of complementary teacher presence can be formulated as a scale, and an educational institution 
must in each case decide the extent to which it will complement the asynchronous teacher presence with 
synchronous presence forms either online or on campus. In this project UCSJ (the provider) has decided to 
supplement the design framework with blended learning activities on campus. Through a series of design 
experiments, the project has therefore developed principles which can complement the overall design 
framework. 

6.4 MOOCs and blended learning 

"Blended learning" is a floating signifier and definitions vary considerably. We use the following definition: 
"Blended learning courses integrate online with face-to-face instruction in a planned, pedagogically valuable 
manner, and do not just combine but trade-off face-to-face with online activity (or vice versa)" (Vignare, 2007 
P-38). 

Blended learning can improve the quality of education (Garrison and Vaughan, 2008), but as in all education, 
this depends on the quality of the instructional design. For instance, using blended learning in addition to a 
MOOC as a compromise to fix a poor online environment or to support the habits of the 'teacher-dependent' 
learner, does not increase the learning effect (Milligan and Ringtved, 2015). 

Research on design principles for combining MOOCs with face-to-face teaching is scarse (Holotescu et a I, 2014; 
Israel, 2015). It is not possible to combine classroom instruction with the original MOOC concept which has a 
large number of participants spread throughout the world. Blended learning is possible only in concepts that 
are not massive, e.g. the so-called "Little Open Online Course" (LOOC), Small Private Online Course (SPOC) 
(Chauhan, 2015), or in concepts combining a group of enrolled students on campus with global participants 
(Ronkowitz & Ronkowitz, 2015). In our project, we work primarily with small MOOCs similar to a SPOC. 


Our design experiments with blended learning have been inspired by the COI Framework (Garisson & 
Anderson, 2011). We have used the latest updated COI model (2015): 


The Community of Inquiry 



https://coi.athabascau.ca 


Figure 2: The COI framework, 2015. 

There are two major differences between this model and the learning environment we have developed in our 
adaptive MOOC. Firstly, the "teaching presence" in a MOOC is mediated (in asynchronous videos) according to 
the design principle referred to above. Secondly, the students 1 "cognitive presence" is not about engagement 
"with content". Participants engage with different content according to their personalized curriculum. This 
means that interactions between teacher and participants on campus must be prioritized in a different way 
than in traditional blended learning concepts. 
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We have developed the following design principles for face-to-face interaction in a blended adaptive MOOC 
concept. 

■ It is not possible to teach common content - but the subject can be introduced. 

In our traditional blended learning concepts, classroom interactions are often given priority to content that 
depends on learning methods, which are not possible to implement online, e.g. learning processes that require 
sensory based perception. This is not possible in an adaptive design in which participants each have their own 
curriculum. But even if the participants already have a range of skills and extensive work experience in a 
profession, the subjects and disciplines develop over time and all students will benefit from having introduced 
the newest paradigms. 

Use face-to-face interaction to support the asynchronous and mediated teacher presence in the MOOC. 

■ Through the introduction of the subject and the course in general, participants establish a trust to the 
teacher as an expert who can get them all the way through the course. Monitor student performance in 
the MOOC and elaborate in face-to-face teaching, on the content that is difficult for all students. 

■ Provide adaptive response in terms of individualized and differentiated feedback on individual performance 
in the MOOC. 

Because the participants have different personalized curriculums, time on campus must be given priority to 
what is absent in a MOOC, i.e. professional feedback from the teacher. In doing so, the student model can be 
updated and the recommendation of an adaptation strategy which took place at the beginning of the course 
may be revised in dialogue with each student. 

■ Scaffold learning in and with a MOOC. 

The learning effect of participating in a MOOC is closely related to whether one has learned how to learn 
through this training format (Milligan and Griffin, 2015). Because most of our participants are MOOC 
beginners, we have to support them in learning in and with a MOOC. 

We also use face-to-face interaction in the same manner as in our other blended concepts. We support "goals 
and direction", we are "setting climate" and we support "discourse", i.e. students' interaction with each other 
and with the content. 

7 Evaluation 

First of all we distinguish between evaluation of the technical aspect of the artefacts developed in the project 
and the usefulness and organizational impact of these artefacts (Hevner & Chatterjee, 2010). Secondly, we 
distinguish between the development of technologies and technical systems per se and the development of 
learning design, enhanced by new technologies. In our research, the evaluation has focused on the latter. 

The evaluation of the developed design principles follows methods and guidelines from Design Based Research 
(Akker et. al., 2006; McKenney & Reeves, 2012). In our evaluation we distinguish between the design principles 
outlined in the framework above, and specific designs for learning developed by individual teachers/authors in 
a given educational, institutionalized context. The specific design which teachers and an educational institution 
develop on the basis of the framework will always be a unique and particular solution to a so-called "wicked 
problem" (Buchanan, 1992). In a DBR project there is no straight line from the developed theory (design 
principles) and the actual design solution. "Design principles are not intended as recipes for success, but to 
help others select and apply the most appropriate substantive and procedural knowledge for specific design 
and development tasks in their own settings" (McKenney et. al. 2006 p 73). 

A specific design will always be a codified representation of the design principles based on their 
interpretations. And even though this project has developed technologies and artefacts to be integrated in the 
learning design (a self-assessment tool and a prototype of the MOOC-platform, Moodle), there is a risk that 
such prototypes will be scaled and used without understandings of the basic theoretical assumptions 
underlying the developed design principles. The developed design principles and artifacts function only as 
"boundary objects" (Star & Griesemer, 1989) between researchers and the teachers/authors. Specific learning 
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designs can therefore easily develop into mutations (legitimate or lethal) which research should subsequently 
study in order to revise the developed theory (Hung et. al., 2010). 

The relationship between design principles and specific designs is illustrated in the following model. 


Research 

Study 


People embodied 
with philosophical 
understandings 



Figure 3: Framework for translation and extension/scaling innovations (Hung et. al., 2010 p. 93) 

The designs developed by teachers/authors have been evaluated according to feasibility, legitimacy and 
efficacy (McKenney, Nieven & Akker, 2006; McKenney & Reeves, 2012). Feasibility relates to the realization of 
the design in an educational institution. Is the design relevant and feasible for the institution? Legitimacy is a 
matter of the compatibility of a design with the basic view of learning, subjects and course design in an 
educational institution. Efficacy relates to the cost-benefit ratio between resources and desired outcomes of 
the design. The degree of feasibility, legitimacy and efficacy affects the intended design an educational 
institution produces and offers to its customers. But the intended design is not the same as the implemented 
and the attained design. The intended design is what the design is set out to do. The implemented design is 
how it is actually used in practice by teachers and students, and the attained design is the specific outcome of 
the design - in our case the learning outcome (Mckenney & Reaves 2012). 

An evaluation design must test both the intended, the implemented and the attained designs which we have 
visualized in the following evaluation model used in the project: 


Design 

principles 


Intended 

design 


Implemented 

design 
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design 


Development of design 
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Explorewhat changes 
must be madeto the 
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V implementation in 
f different contexts. 


Figure 4: Evaluation model 

The evaluation was conducted using alpha testing, beta testing and in the year to come also gamma testing 
(Mckenney & Reeves, 2012). Our alpha trials have been controlled by the research team with maximum 
support for teachers and students. The aim was to test the feasibility of the design in our institution and 
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explore the teachers' and the students' assumptions about viability and impact on learning outcome. In our 
beta test we have tested designs in a real life context but still with some support. The goal has been to explore 
conflicts between the intended design and its implementation according to institutional feasibility and 
viability, map out fostering and hindering conditions for implementation and measure the initial impact on 
learning outcomes. Gamma testing are trials focusing on widespread adoption with minimal support. The 
purpose of this kind of evaluation is to measure the impact, the efficiency and the sustainability of the design. 
What is the relationship between learning outcomes among participants, the use of resources and fostering 
and hindering conditions for sustainability? The project is in a planning phase for gamma testing. 

7.1 Data sources and analysis 

We have collected data and conducted qualitative analyses of: 

■ Design workshops with MOOC designers (the teachers/authors). 

■ Content and design of the MOOCs produced by the institution on a Moodle platform. 

■ Interviews with MOOC designers (the teachers/authors). 

■ The institution's decisions about strategy, management and support according to policy papers from the 
project. 

■ Observations of student self-assessments and additional training on campus. 

■ Student interactions in the MOOC. 

■ Interviews with students and MOOC teachers. 

■ Results from a questionnaire survey on user satisfaction conducted by the institution. 

Data were analyzed in order to saturate categories for understanding (Glaser & Strauss, 1967/2008). In the 
evaluation of the implemented design we were looking for categories in terms of "mechanisms" (Pawson & 
Tilley, 1997). In our understanding of the relationship between the intended design, the implemented design 
and the attained design, we are inspired by "realistic evaluation" as illustrated in the model below. 


The implemented design 





Figure 5: Relationships between design levels. (Inspired by Pawson & Tilley, 1997/2013 p. 58) 

Particularly in the analysis of the relevance of the design and its usefulness to students, categories were 
consistent with the categories developed by Everett Rogers (Rogers 1962/1995) in his theory about attributes 
of innovations and their rate of adoption. These categories were therefore applied to the study. 

7.2 Results 

7.2.1 The institution's choice of intended design 

The framework has been realized as follows: University College Zealand (UCSJ) has produced MOOCs in seven 
different subjects and will produce MOOCs for all subjects in the Danish primary school by 2018.The first step 
for a participant in a MOOC course is to complete a self-assessment in which he assesses his qualifications on a 
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scale from 1- 5 on the basis of the learning objectives of the course - objectives formulated in terms of 
competencies. The self-assessment is conducted with a tool developed within the project which based on the 
student's input generates and visualizes a competence profile illustrating the percentage of the curriculum 
that the student must study. The developed tool also works as a recommendation and navigation system that 
generates an adaptive match between the competence gap and selected study themes. The MOOC platform 
for each of the 7 subjects is designed so that adaptation is possible, no matter which skills the student needs 
to acquire to pass the exam. The curriculum in a subject is divided into a number of themes that are organized 
so they can be accessed without progression. Each theme is assigned a set of competencies that the students 
can acquire through study work e.g. by accessing video resources, texts, exercises and quizzes and by 
participating in peer to peer response and collaboration etc. The recommendation system is supplemented by 
a 90 minutes guidance session with a teacher in order to further support the technical recommendation 
system. During the guidance session, the student's self-assessment is reviewed and the teacher provides 
additional guidance on how to choose from the adaptive themes and navigate within the MOOC platform. 

The evaluation shows that there is a high degree of fidelity between the design principles and the intended 
design of the 7 MOOCs. 

As UCSJ has decided to offer a blended learning concept for teacher professional development, the students 1 
interaction in the MOOC is supplemented by face to face instruction on campus. The extent of on-campus 
training is chosen by the municipalities who are given the opportunity to purchase this kind of additional 
training. In relation to this, two interesting mutations have been identified in that one of the municipalities has 
decided not to buy additional training, whereas another municipality has decided to purchase additional 
training to such an extent that the concept can be characterized as "MOOC enhanced classrooms". 

The intended design is a result of decisions at three different organizational levels: the strategic, the tactical 
and the pedagogical levels. At the strategic level UCSJ has decided an entirely new business model for 
continuing education. MOOC production of courses is both costly as well as time-consuming and while 
municipalities pay for additional on-campus teaching, UCSJ has to face production costs a year in advance of 
receiving revenue from the courses. On the tactical level UCSJ has employed people for video production, 
organized training and support to teachers who produce MOOC content, organized workflow of teachers in a 
new way and initiated negotiations with the teachers on the copyright of MOOC content. 

The evaluation shows that the intended design is feasible but its efficacy has been questioned by 
management. However, as UCSJ for strategic reasons and in accordance with policy papers has decided to be 
at the forefront of technology-enhanced education in Denmark, the project is still carried out on a large scale. 

The strategic and tactical decisions are organizational prerequisites for the realization of an innovative learning 
design. On a pedagogical level the intended design is a result of the teachers 1 decisions and choices of learning 
design for the 7 MOOCs. The evaluation shows the following: 

■ There is a relationship between the discipline or subject matter and the teachers' choices of MOOC 
pedagogy, but a subject does not determine a specific MOOC-pedagogy and there are more similarities 
than differences between the seven MOOCs. Even though the framework focuses on the interpretation of 
how content is situated in a specific subject culture, a stronger emphasis of this must be done in future 
introductions of the framework. 

■ The greatest legitimacy problem of the design is related to the breakdown of a subject into competency 
units without progression. This design element conflicts with the teachers 1 common conceptions of 
curriculum design. This key constraint has to be introduced in a better way for MOOC designers. This can 
be done by producing additional "boundary objects" in terms of exemplary materials, which can support 
the designers 1 professional curriculum reflections and show how a competence-oriented curriculum can 
be transformed into a MOOC (van den Akker, 1998). 

■ Teachers who teach the same subject or family of subjects may have very different views on the subject, 
which is reflected in their learning design. We have data showing that in some disciplines there are very 
different views on the MOOC pedagogy; some teachers believe that you cannot teach their subject in a 
MOOC, while others believe that it can easily be done. The evaluation has identified a conflict between 
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legitimacy and feasibility as UCSJ in some disciplines has had difficulty finding teachers for MOOC- 
production. 

■ Teachers have different approaches to the design process and their approaches may change during the 
design process. We have observed transformation strategies where on-campus instruction is simulated or 
remediated in the learning design of the MOOC. If the teacher is a beginner when it comes to MOOC 
design, or receives little pedagogical support, our data indicate that the teacher will employ a simulation 
strategy. For the experienced MOOC or e-learning designer, a MOOC offers a completely different learning 
environment with specific affordances and constraints, which is reflected in their design. 

■ Design choices based on an explicit theory of learning is not observed in the 7 MOOCs. Design choices 
related to learning processes are very similar in the seven MOOCs which all share many similarities with X- 
MOOCs. This may be due to the fact that learning theory is not explicitly reflected within the framework 
and the associated design principles. The design framework should at this point be revised in future 
iterations. 

7.2.2 The implemented and attained design 

The evaluation of the implemented design shows that there are four different clusters of mechanisms which 

have significance for the attained design in different local contexts: 


■ The participants' perception of relevance and usefulness of the intended learning design. 

* UCSJs introduction of the intended design for students. 

* The students'study conditions. 

* The students' academic qualifications. 


In relation to the implementation of the intended design, our data indicate that the principle of a personalized 
curriculum and an adaptive learning design with options for personalized learning paths works well for 
students. The students are very satisfied with the content in the MOOC and they assess learning resources to 
have high professional quality. Data show that the students spend most time accessing learning resources 
while peer to peer response activities and assignments have lower priority. The data also indicate that 
additional teacher presence is an essential mechanism for students 1 experience of the legitimacy of the design. 
The students prefer a design with additional training on campus, and the mutation described above as a 
"MOOC-enhanced classroom" has been well accepted. 

In contrast, this mutation is seen as a major challenge by the MOOC teachers since it is difficult to organize 
meaningful teaching on campus for students who each have their own curriculum. The development of 
additional design principles for blended learning has not been sufficient to support the teachers in meaningful 
planning of activities on campus and the framework must be improved in future iterations. 

The students 1 experience of the relevance of the design and its usefulness can be explained by five attributes 
of the design (Rogers 1962/1995): 

The degree to which the design is perceived as consistent with students' existing values 
and past experiences regarding ongoing education. 

■ The degree to which the design is perceived as relatively difficult to understand and use. 

■ The degree to which the potentials of the design can be observed or explained for students. 

■ The degree to which designs may be experimented with on a limited basis. 

■ The degree to which the design is perceived as being better than the ongoing education the 

students are used to. 

Data from interviews and observations show that the design is very far from students 1 experience with ongoing 
education and teacher professional development. Particularly the lack of teacher presence in the MOOC and 
especially the lack of teacher feedback is considered challenging. Peer to peer response activities are not 
perceived as a qualified replacement of teacher feedback, and the MOOC teachers have had difficulty 
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explaining why this kind of feedback is meaningful, it has also been difficult for the students to learn how to 
use this part of the design, which may have contributed to the fact that this type of activity is largely 
deselected. 

However, the overall design principle (the affordance of the design) has not been difficult to observe as an 
attribute to the design or explain to students. They are all experienced school teachers who appreciate the fact 
that the design allows for a personalized curriculum and an adaptive personalized learning path for each 
student although it should be noted that the relative advantage of this does not imply that students simply 
accept the perceived constraints. 

Finally, students have been unable to test in advance aspects of the design as the concept has been fully 
implemented from day one without an initial introduction or training in the study skills required of learners in 
a MOOC. In line with other studies, we find that MOOC learning skills are important for learning outcomes 
(Milligan & Griffin, 2015), and the MOOC design needs to scaffold and develop these skills. The framework 
must therefore be improved in future iterations, especially in relation to introduction of peer to peer response 
activities and collaborative community participation. 

According to student evaluations there is furthermore an unused potential in relation to community 
participation. The framework (Figure 1) visualizes how students are rooted in a personalized learning network 
(PLN), but this part of the framework has never been integrated into either the intended or implemented 
design. UCSJ has only used the framework to implement SPOCS (Small Private Online Courses), which are only 
open to students whose municipality has purchased access. Several students have identified an even greater 
potential for ongoing teacher professional development if the MOOCs were open to all teachers in a given 
subject within a given municipality or, alternatively, across municipalities. Firstly, the student's would like to 
have access to the MOOC after they have finished their exams in order to keep their professional skills up-to- 
date. Secondly, the students' would like to collaborate in the MOOC with those of their colleagues who already 
have a formal exam in a subject. Colleagues, who also would like continuous professional development and 
access to the latest knowledge in a subject, knowledge which the MOOC could represent through regular 
updates. The MOOC could thus become a professional community for ongoing teacher professional 
development in a municipality - and not just a training course. The evaluation has identified a new "legitimate 
mutation" of the design that may increase the relative advantage of the design considerably. This could 
significantly affect the students 1 perception of the design, and the mutation may also be a new business case 
for UCSJ. 

However, the most important mechanisms affecting the students' experience of the usability of the design, do 
not at all relate to the intended design or the way it has been implemented by UCSJ. The most saturated 
category in our evaluation concerns the study conditions. In Denmark, it has been customary that the 
employer pays for the time employees spend on in-service training activities on campus. The basic design 
framework, however, can be realized in a design which can be accessed online via asynchronous activities, and 
this design has only to a limited extent been supplemented with activities on campus in a blended learning 
concept. The flexibility of the design in time and space has been important for the local viability of the design. 
Some municipalities have chosen to implement the design in their local context in a way in which the 
employees get very little or no time to participate in study activities. The consequence is that several students 
see the concept as a discount solution for teacher professional development, and the intended design is in this 
local context mutated to a "lethal mutation". 

The last category, which affects the learning outcome, is the students 1 academic qualifications. We have so far 
only few data regarding this as only a small group of students have taken exams, but based on interviews with 
MOOC teachers, we have formulated a preliminary hypothesis regarding the correlation between the students 1 
academic prerequisites and the learning outcomes measured in a test. 

We differentiate between the students 1 academic qualifications in a subject and their educational 
qualifications to teach a subject in school. Based on these two types of academic skills, we have generated a 
model visualizing four different profiles of school teachers who need formal supplementary education: 
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Figure 6: Student profiles and academic prerequisites 
The evaluation shows the following: 

■ The intended and implemented designs from UCSJ are equivalent to the needs of the students from 
Quadrants 1-3, who all have different personalized curriculum needs. 

■ It is possible to pass the exam based on qualifications from Quadrants 1-3. 

■ The design gives students from Quadrant 2 in particular the opportunity to demonstrate their excellence. 

■ Students who have poor academic qualifications in a subject and poor pedagogical qualifications 
(Quadrant 4) cannot pass the exam and should be advised to take an ordinary education with full 
curriculum. 

8 Further research 

In the forthcoming year, we have planned a research project with tree different activities: 

a) Redesign of the developed framework 

According to the evaluation of the design of an adaptive MOOC we need to develop a new iteration of the 
framework. First of all a community and collaboration-oriented view on learning and participation should be 
better integrated into the framework. And secondly our design principles for additional blended learning 
activities need to be improved. The evaluation of the framework also showed that the "relationships" and the 
"principles" in the framework have to be better introduced to the MOOC designers and the MOOC teachers. 
We have therefore decided to produce exemplary materials which can introduce the framework. The 
evaluation shows that particularly three areas have to be better introduced. First of all the exemplary material 
must show examples of a competence-oriented curriculum design. Secondly the material must show examples 
of different collaborative learning activities in the MOOC - especially peer to peer activities. Thirdly the 
material must show examples of meaningful blended learning activities on campus. 

b) Gamma testing of the design 

In the year to come we have planned a gamma testing of the design. Our objective is to study the relations 
between the implemented design in different context, the participants' academic qualifications and their 
learning outcomes. 

c) Exploring innovative mutations. 

The evaluation has identified two promising but unexpected mutations of the design and the research project 
will study and develop learning principles for these mutations. First of all the evaluation has identified a 
potential for teacher professional development for all teachers in a municipality, if the design is transformed 
from an in service course to an open learning community. We have therefore decided to develop an entirely 
new framework for producing MOOCs for professional community building and ongoing teacher professional 
development for all teachers in a municipality. Secondly the evaluation has showed a potential for using our 
archived MOOCs in a blended learning concept in our ordinary bachelor degree programs. We are therefore in 
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the planning phase of a new research project, that in the next two years will explore and development a 
framework to guide the use of MOOCs in "MOOC enhanced classrooms" in our ordinary undergraduate 
program. 
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